Papers by Gustav Eje Henter
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches for positional dependencies do not satisfy all criteria for optimal position encoding. |
| Approach: | They propose a translation-invariant self-attention approach that accounts for relative position between tokens in an interpretable fashion without conventional embeddings. |
| Outcome: | The proposed model improves on regular ALBERT on GLUE tasks while adding orders of magnitude less positional parameters. |